{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 程序说明\n",
    "名称：使用sklearn wrapper做参数搜索\n",
    "\n",
    "时间：2016年11月17日\n",
    "\n",
    "说明：建造一个简单的卷积模型，通过使用sklearn的GridSearchCV去发现最好的模型。\n",
    "\n",
    "数据集：MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.加载keras模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "np.random.seed(1337)  # for reproducibility\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers import Convolution2D, MaxPooling2D\n",
    "from keras.utils import np_utils\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from sklearn.grid_search import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.变量初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nb_classes = 10\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load training data and do basic data normalization\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)\n",
    "X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "X_train /= 255\n",
    "X_test /= 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 转换类标号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# convert class vectors to binary class matrices\n",
    "y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "y_test = np_utils.to_categorical(y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.建立模型\n",
    "### 使用Sequential（）\n",
    "构造一个有两个卷积层和若干个全连接层组成的模型，这里全连接的层数是由参数所决定的。\n",
    "\n",
    "dense_layer_sizes：层尺寸的列表。这个列表中对于每个层都有一组数字。\n",
    "\n",
    "nb_filters：每个卷积层中滤波器的个数\n",
    "\n",
    "nb_conv：卷积核的尺寸\n",
    "\n",
    "nb_pool：用于max pooling的池化面积"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_model(dense_layer_sizes, nb_filters, nb_conv, nb_pool):\n",
    "    '''Creates model comprised of 2 convolutional layers followed by dense layers\n",
    "    dense_layer_sizes: List of layer sizes. This list has one number for each layer\n",
    "    nb_filters: Number of convolutional filters in each convolutional layer\n",
    "    nb_conv: Convolutional kernel size\n",
    "    nb_pool: Size of pooling area for max pooling\n",
    "    '''\n",
    "\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                            border_mode='valid',\n",
    "                            input_shape=(img_rows, img_cols, 1)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "    model.add(Dropout(0.25))\n",
    "\n",
    "    model.add(Flatten())\n",
    "    for layer_size in dense_layer_sizes:\n",
    "        model.add(Dense(layer_size))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                  optimizer='adadelta',\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.sklearn接口\n",
    "\n",
    "KerasClassifier()实现了sklearn的分类器接口\n",
    "\n",
    "`keras.wrappers.scikit_learn.KerasClassifier(build_fn=None, **sk_params）`\n",
    "\n",
    "build_fn:可调用的函数或类对象\n",
    "\n",
    "sk_params:模型参数和训练参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dense_size_candidates = [[32], [64], [32, 32], [64, 64]]\n",
    "my_classifier = KerasClassifier(make_model, batch_size=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sklearn中的GridSearchCV函数\n",
    "说明：对估计器的指定参数值进行穷举搜索。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "validator = GridSearchCV(my_classifier,\n",
    "                         param_grid={'dense_layer_sizes': dense_size_candidates,\n",
    "                                     # nb_epoch可用于调整，即使不是模型构建函数的参数\n",
    "                                     'nb_epoch': [3, 6],\n",
    "                                     'nb_filters': [8],\n",
    "                                     'nb_conv': [3],\n",
    "                                     'nb_pool': [2]},\n",
    "                         scoring='log_loss',\n",
    "                         n_jobs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 开始拟合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python2.7/site-packages/ipykernel_launcher.py:13: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(8, (3, 3), padding=\"valid\", input_shape=(28, 28, 1...)`\n",
      "  del sys.path[0]\n",
      "/opt/anaconda/lib/python2.7/site-packages/ipykernel_launcher.py:15: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(8, (3, 3))`\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "40000/40000 [==============================] - 8s - loss: 0.8491 - acc: 0.7111     \n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.5334 - acc: 0.8272     \n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4470 - acc: 0.8591     \n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4037 - acc: 0.8718     \n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3755 - acc: 0.8813     \n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3489 - acc: 0.8904     \n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3274 - acc: 0.9005     \n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3082 - acc: 0.9030     \n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3003 - acc: 0.9052     \n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2814 - acc: 0.9111     \n",
      " 3648/20000 [====>.........................] - ETA: 0s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python2.7/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19264/20000 [===========================>..] - ETA: 0sEpoch 1/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.9092 - acc: 0.6986     \n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.6375 - acc: 0.7951     \n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.5235 - acc: 0.8334     \n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4263 - acc: 0.8644     \n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3795 - acc: 0.8807     \n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3409 - acc: 0.8923     \n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3216 - acc: 0.8991     \n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3044 - acc: 0.9061     \n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2910 - acc: 0.9086     \n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2794 - acc: 0.9123     \n",
      " 3584/20000 [====>.........................] - ETA: 0s- ETA: "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python2.7/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18848/20000 [===========================>..] - ETA: 0sEpoch 1/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.9082 - acc: 0.6951     \n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.5760 - acc: 0.8113     \n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4753 - acc: 0.8479     \n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4250 - acc: 0.8639     \n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3844 - acc: 0.8766     \n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3588 - acc: 0.8850     \n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3382 - acc: 0.8910     \n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3283 - acc: 0.8975     \n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3071 - acc: 0.9012     \n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2958 - acc: 0.9068     \n",
      " 3488/20000 [====>.........................] - ETA: 0s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python2.7/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19872/20000 [============================>.] - ETA: 0sEpoch 1/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.8452 - acc: 0.7197     \n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.5408 - acc: 0.8256     \n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4448 - acc: 0.8586     \n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3964 - acc: 0.8776     \n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3534 - acc: 0.8889     \n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3236 - acc: 0.9004     \n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2984 - acc: 0.9085     \n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2840 - acc: 0.9107     \n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2731 - acc: 0.9152     \n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2623 - acc: 0.9183     \n",
      " 2304/20000 [==>...........................] - ETA: 1s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python2.7/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19552/20000 [============================>.] - ETA: 0sEpoch 1/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.7876 - acc: 0.7346     \n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4913 - acc: 0.8419     \n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4160 - acc: 0.8694     \n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3731 - acc: 0.8836     \n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3507 - acc: 0.8919     \n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3256 - acc: 0.8998     \n",
      "Epoch 7/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3127 - acc: 0.9042     \n",
      "Epoch 8/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2996 - acc: 0.9098     \n",
      "Epoch 9/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2879 - acc: 0.9102     \n",
      "Epoch 10/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.2772 - acc: 0.9151     \n",
      " 2304/20000 [==>...........................] - ETA: 1s"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda/lib/python2.7/site-packages/sklearn/metrics/scorer.py:137: DeprecationWarning: Scoring method log_loss was renamed to neg_log_loss in version 0.18 and will be removed in 0.20.\n",
      "  sample_weight=sample_weight)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19360/20000 [============================>.] - ETA: 0sEpoch 1/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.8804 - acc: 0.6992     \n",
      "Epoch 2/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.5414 - acc: 0.8229     \n",
      "Epoch 3/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4585 - acc: 0.8522     \n",
      "Epoch 4/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.4067 - acc: 0.8681     \n",
      "Epoch 5/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3634 - acc: 0.8824     \n",
      "Epoch 6/10\n",
      "40000/40000 [==============================] - 6s - loss: 0.3378 - acc: 0.8927     \n",
      "Epoch 7/10\n",
      "36128/40000 [==========================>...] - ETA: 0s - loss: 0.3205 - acc: 0.9000"
     ]
    }
   ],
   "source": [
    "validator.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 打印最好模型的参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The parameters of the best model are: \n",
      "{'dense_layer_sizes': [64, 64], 'nb_conv': 3, 'nb_pool': 2, 'nb_epoch': 6, 'nb_filters': 8}\n"
     ]
    }
   ],
   "source": [
    "print('The parameters of the best model are: ')\n",
    "print(validator.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 返回模型\n",
    "\n",
    "validator.best_estimator_ 返回sklearn-wrapped版本的最好模型\n",
    "\n",
    "validator.best_estimator_.model 返回（unwrapped）keras模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s     \n",
      "\n",
      "\n",
      "loss :  0.0535527251991\n",
      "acc :  0.9825\n"
     ]
    }
   ],
   "source": [
    "best_model = validator.best_estimator_.model\n",
    "metric_names = best_model.metrics_names\n",
    "metric_values = best_model.evaluate(X_test, y_test)\n",
    "print('\\n')\n",
    "for metric, value in zip(metric_names, metric_values):\n",
    "    print(metric, ': ', value)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  },
  "ssap_exp_config": {
   "error_alert": "Error Occurs!",
   "initial": [],
   "max_iteration": 1000,
   "recv_id": "",
   "running": [],
   "summary": [],
   "version": "1.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
